Towards an Assessment of an AI System ' s Validity by a

Although there seems to be no (formal) way of proving the validity of an AI system, the authors present some ideas on developing a validity statement based on a Turing-test methodology with a set of "good" 1 test cases. The solution of these test cases will be rated by a panel of expert validators. The methodology is called the Turing-test, because a random process of distributing the test case with solutions to the diierent valida-tors ensures that no validator knows who the author of a test case solution is. The objective of this is, of course, to make the result of the validation process (the validity statement) more objective. Furthermore, in an eeort to maximize objectivity , the approach described here includes a competence scale for each validator. This is done for each test case separately due to the fact that competence is a property of an expert , which normally isn't distributed homogeneously within the "input space" of the AI system. The degree of competence is estimated by considering the experts' behavior while solving the test cases and rating the test case solutions. The main sources of this competence esti-1 "Good" in this context is the subject of other papers published by the authors and their group. mation are the experts' own competence assessment , their behavior while rating the own solution, and the rating of their solution by other experts.